CONTEXT
What's Sigma, and what's duplicate rows?
What's Sigma?
Sigma is Excel on steroids: the familiarity of Excel with the power of BI tools
Sigma is next-generation analytics and business intelligence that scales billions of records using spreadsheets, SQL, Python, or AI—without compromising speed and security. It allows users to perform complex data analysis with an easy-to-use interface.
Why are there duplicate rows?
Result of table joins and others
When joining two tables of different granularity, rows may be duplicated, especially if join keys are repeated differently in each table. However, duplicates can also occur due to other factors, not just joins.
See the video below as as example of how this is done today.
PROBLEM STATEMENT
How might we design a remove duplicate user flow that’s intuitive and does not require extensive knowledge of Sigma?
It should be an easy task, only if users know how to use "Groupings" 🧐
Workaround
Users can manually dedupe using "grouping"
Group the columns being compared for duplicates, add a new rowNumber column, filter for ‘1’ to keep the first rows, and add a sort for a deterministic result.
Problem with workaround
But it's a tedious and confusing process
Users don’t realize that grouping can solve their issue, and they go to support engineers because they can’t find a "remove duplicates" feature.
Configuring individual column calculations is inefficient when dealing with many columns.
Sales Associate @ Blackstone:
"Is there a way to remove duplicate rows in a table, similar to how Excel handles it?"
Business Development Manager @ REI:
"After joining two tables, I ended up with a lot of duplicate rows. I’ve heard using groupings can fix this, but it seems pretty complicated and I couldn't figure it out. Can we set up a meeting and get it fixed? Ty"
IMPACT
Removing duplicates is an impactful & the most requested feature according to the Jira ticket
>32
Unique customers impacted
This includes some of our biggest customers such as BlackStone and Colgate.
92%
Reduction in No. of clicks
Easy-to-use module that doesn’t require users to understand formulas or grouping.
84%
Reduction in action time
This action now takes seconds instead of over a minute with the new design.
PROCESS
I researched, prototyped, tested, iterated (and solved unexpected problems)
🔎 Research
I researched how products like Excel, Looker, and Power BI implement this feature
✍🏻 Prototype
I reviewed similar features in our product and consulted with engineers for feasibility
🧪 User Testing
I tested with non-engineers in the office, then with end users for unbiased feedback
🔧 Iteration
Based on feedback, I iterated the copy and expected results before handing off to engineers
This feature stems from requests by novice users
Snapshot of a non-technical Sigma users
Primary function might not be working with data
Likely familiar with Excel
Uses Sigma once or twice a week
New and learning to use Sigma Computing for more advanced data analysis and reporting.
Their goal is to have something similar to excel
Quickly create child tables for specific data views without disturbing the original dataset.
Ensure tables maintain consistent column orders, as she finds unpredictable changes disorienting.
Easily make edits to tables through a clear, accessible interface.
Iterations
3 parts: entry point, configuration modal, and result
I formed my opinion and brought my designs to the team at an early stage. Click on the images to zoom in!
User Testing 🔁 Iteration
I did in-office and user testing with end users, gathering key insights for three design components.
Conducted ad-hoc in-office testing
Spoke directly to 4 end users from 2 customers, which was highly beneficial
Created a comprehensive testing script to represent the Sigma design team well
Key points emerged for the three design components during the testing process
Where would users access this feature?
Insight 1: What’s technically and logically correct might not always align with user expectations
Before Iteration
The feature entry point was placed at the element level because technically any action that affects the entire table belongs to the element menu.
After Iteration
Users expect to find it in the column menu, even though it is a element-level action.
There eyes movement is focued on the duplicated rows rather than the top Element menu, which is not easily discoverable.
How do users configure this action? What decisions are made?
Insight 2: Using “defaults” to reduce clicks and cognitive load is not ideal for such a technical feature.
Before Iteration
The conversation style is effective and clear to users.
The modal design was long and many clicks are required.
No clear communication about the expected result.
Sorting order was not explicitly requested when selecting default, leading to non-deterministic results.
After Iteration
The modal design is shorter and more efficient.
Fields are auto-populated to minimize clicks, acknowledging user preference to edit steps 2 and 3.
A child table is always created to avoid disrupting groupings, and a tip is added to inform users about the child table creation.
Sorting order is now a requirment to ensure deterministic results.
What would users see as a result of the action?
Insight 3: We don’t challenge novices to learn Sigma; we show the standard result just like in Excel.
Before Iteration
The table is the same as if the user manually created a child from the original and completed all the steps.
Users found the sudden change in column placement due to grouping to be jarring.
After Iteration
The child table is now an intermediate, hidden step to prevent disruption of the workflow and confusion for users.
The final result is a ‘grandchild’ table—flat with fixed column orders.
Users can edit by going to the element source and re-accessing the modal.
FINAL DESIGN
