Frustration-induced complexity in order-disorder transitions of the J1-J2-J3 Ising model on the square lattice

We revisit the field-free Ising model on a square lattice with up to third-neighbor (NNNN) interactions, also known as the J1-J2-J3 model, in the mean-field approximation. Using a systematic enumeration procedure, we show that the region of phase space in which the high-temperature disordered phase is stable against all modes representing periodic magnetization patterns up to a given size is a convex polytope that can be obtained by solving a standard vertex enumeration problem. Each face of this polytope corresponds to a set of coupling constants for which a single set of modes, equivalent up to a symmetry of the lattice, bifurcates from the disordered solution. While the structure of this polytope is simple in the half-space J3>0, where the NNNN interaction is ferromagnetic, it becomes increasingly complex in the half-space J3<0, where the antiferromagnetic NNNN interaction induces strong frustration. We then pass to the limit N→∞ giving a closed-form description of the order-disorder surface in the thermodynamic limit, which shows that for J3<0, the emergent ordered phases will have a "devil's surface"-like mode structure. Finally, using Monte Carlo simulations, we show that for small periodic systems, the mean-field analysis correctly predicts the dominant modes of the ordered phases that develop for coupling constants associated with the centroid of the faces of the disorder polytope.