The straight line lateral limits were chosen partly for simplicity, since the lateral limit is constant regardless of the dip of the slope or failure plane. With the curved lateral limits (defined by a cone) the limits are a function of the dip of the slope.
Both methods are in use and it seems that straight line limits have become more commonly used. The critical area for both approaches is very similar and under most circumstances the number of critical poles will be nearly identical. The reason for imposing lateral limits is due to the observation that planar (and toppling) failure tends to occur when the dip direction of the failure plane is within a certain angular range of the slope dip direction. This limit is empirical and values between plus/minus 15 and 30 degrees have been proposed (e.g. Goodman, Hudson, and Harrison). The kinematic analysis module in Dips is heavily based on Chapter 18 of “Engineering Rock Mechanics” by John Hudson and John Harrison. The book outlines an approach based on linear lateral limits.
Another (minor) reason for not using the curved limits, is that the straight line limits can be used with either pole vectors or dip vectors and the results are identical. With curved limits, the dip vector and pole vector results are not equivalent, unless you convert the curved limits when using dip vectors. Since Dips can do kinematic analysis on both poles and dip vectors we choose to use linear lateral limits.
Practically speaking, there is not a great deal of difference whether you use straight or curved limits. The curved lateral limits cover a larger area and could be considered more conservative.
Another consideration is the following. For planar sliding, the difference between straight and curved limits becomes more pronounced as the dip angle of a failure plane decreases. (see image below, highlighted yellow regions). However, the probability of oblique planar failure also decreases as planes become flatter. Therefore, we reason that poles which plot in these yellow regions represent a low probability of failure. Conversely, as you approach the stereonet perimeter, the straight and curved lateral limits converge to the same point. Poles plotting near the perimeter (steeply dipping planes) are of the most concern and have the highest probability of oblique failure. For this case, the straight and curved limits are virtually identical (e.g. consider vertical or near vertical slopes).
For flexural toppling, the logic is the same. Since the critical poles are near the perimeter, there is virtually no difference between the use of straight or curved limits.
In general, we feel that straight line limits are sufficient for most analyses and are easier to understand and implement.
You can still draw a curved lateral limit by hand using the cone tool. You need to use a pole vector plot for this. For a lateral limit of X degrees, you should draw a cone with radius of 90-X degrees, centered at a trend offset 90 degrees from the dip direction of the slope and a plunge of 0. For example, if you are performing a kinematic analysis with 20 degree lateral limit and a slope with dip/dip direction of 75/230 you would place a cone with radius 70 at trend/plunge of 140/0.
You will need to either use the planar sliding failure mode with lateral limits and manually count the number of poles that fall in the space between the linear and curved limits, adding them to the total, or use planar sliding failure mode without lateral limits and count the number of poles that fall inside the total critical area but outside the curved limits, and subtract them from the total.