That is a very good point. It is true that the Hoek-Brown failure criterion is only applicable to jointed rock masses, and is not applicable to failure along a single joint plane. However, the failure surface in RocPlane does not necessarily have to represent a single discrete joint plane. The use of the Hoek-Brown (or Generalized Hoek-Brown) criterion in RocPlane should be considered as follows: the failure plane represents a linear failure surface through a jointed rock mass, rather than a single joint plane. Only under these conditions should the Hoek-Brown failure criterion be applied in RocPlane.
See the Probability of Failure topic.