The Belgian world No. 19 completed the victory in 78 minutes on Wednesday, ending Gauff's unbeaten record against her after five previous meetings.
Mertens ends Gauff's run
Gauff entered the last-16 contest as the highest-ranked player remaining in the women's draw, but struggled to find her rhythm against a determined Mertens.
The American committed eight double faults and made 34 unforced errors as Mertens capitalised on a series of misjudged shots to claim her first victory over Gauff.
Mertens kept her composure when Gauff attempted to fight her way back into the contest and sealed the match with an ace on her second match point.
The victory sends the 30-year-old into the quarter-finals of a WTA 1000 tournament, while Gauff's hopes of another deep run in Beijing are over.
RECOMMENDED STORIES
• Argentina 3-0 Benin: Messi scores and assists twice in emotional farewell to international football
• Super Eagles Leave Nizhny Novgorod for Moscow Straight after the Draw
• Maple Leafs 5-4 Predators: Gavin McKenna scores first NHL goal as Toronto complete dramatic comeback
• Russia 3-3 Nigeria: Super Eagles fight back from two goals down in six-goal thriller
Gauff's difficult few days
The defeat came a day after Gauff spoke publicly about the racist abuse she received online following her previous-round victory over Chinese teenager Sun Xinran.
During that match, Gauff stopped play after noticing flickering LED advertising boards. Following a review, the umpire ordered the point to be replayed, prompting criticism online.
Gauff subsequently condemned the racist messages directed at her and said the volume of abuse had been "insane". The WTA also expressed its support for the American and condemned the racial abuse.
After losing to Mertens, Gauff admitted that the experience had been draining, although she also took responsibility for her performance and said she simply did not play well.
Swiatek awaits in the quarter-finals
Mertens will now face eighth seed Iga Swiatek for a place in the China Open semi-finals.
Swiatek advanced after defeating American teenager Iva Jovic 6-1, 6-3 in one hour and 35 minutes. The Polish star controlled the match from the outset and will now meet Mertens in what promises to be a major quarter-final clash.
For Mertens, the victory over Gauff represents a significant breakthrough after losing her first five meetings with the American. She will now look to carry that momentum into her quarter-final against the former world No. 1.