Revealing the Phase Diagram of Kitaev Materials by Machine Learning: Cooperation and Competition between Spin Liquids

Kitaev materials are promising materials for hosting quantum spin liquids and investigating the interplay of topological and symmetry-broken phases. We use an unsupervised and interpretable machine-learning method, the tensorial-kernel support vector machine, to study the classical honeycomb Kitaev-$\Gamma$ model in a magnetic field. Our machine learns the global phase diagram and the associated analytical order parameters, including several distinct spin liquids, two exotic $S_3$ magnets, and two modulated $S_3 \times Z_3$ magnets. We find that the extension of Kitaev spin liquids and a field-induced suppression of magnetic orders already occur in the large-$S$ limit, implying that critical parts of the physics of Kitaev materials can be understood at the classical level. Moreover, the two $S_3 \times Z_3$ orders exhibit spin structure factors that are similar to the ones seen in neutron scattering data of the spin-liquid candidate $\alpha$-$\mathrm{RuCl}_3$. These orders feature a novel spin-lattice entangled modulation and are understood as the result of the competition between Kitaev and $\Gamma$ spin liquids. Our work provides the first instance where a machine detects new phases and paves the way towards developing automated tools to explore unsolved problems in many-body physics.