In PySpark, a DataFrame with duplicate column names can cause ambiguous and unexpected results with join, transformation, and data
retrieval operations, all while making the code more confusing. For example:
- Column selection becomes unpredictable:
df.select("name") will raise an exception
- Joins with other DataFrames may produce unexpected results or errors
- Saving to external data sources may fail
Case-insensitive duplicates, for example a column named "name" and "Name", are also flagged. This is because having column names that differ only
in casing creates confusion when referencing columns and makes code harder to understand and maintain leading to subtle bugs that are difficult to
detect and fix.
