This page contains methods for performing various intersection tests. Although it does not have an entry for ray vs. line segment intersection, I tried the suggested ray vs. ray intersection test (page 782 of RealTime Rendering 3rd Edition) and it did not work in my case.
I looked around quite a bit and based on an adaptation of this answer, I finally found a method that works fine. Given a ray (with a start point, an end point and direction) and a line segment with the defined start and end points, we perform the 3D line intersection test. However, note that in various graphics APIs, there is always some error when converting screen points to lines (because there is no 11 mapping from screen pixels to exact 3D points in space). Therefore, when performing intersection tests, some degree of tolerance must be considered. This is especially important in the early rejection step of the intersection test algorithm. The early rejection test, checks to see whether the two 3D lines are coplaner. If they aren’t, then no intersection will be reported.
After finding the intersection point, we must check and see if this point lies between the start and end points of the line segment. This can be done by comparing the length of the line segment with the sum of distances of the intersection point from the start point and end point of the line segment respectively. If the length is “almost” equal, then it means that the original ray will intersect with the line segment.
Here is the pseudocode of the described process (adapted from the original answer):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

const double coPlanerThreshold = 0.7; // Some threshold value that is application dependent const double lengthErrorThreshold = 1e3; bool intersection(Ray ray, LineSegment segment) { Vector3 da = ray.End  ray.Origin; // Unnormalized direction of the ray Vector3 db = segment.End  segment.Start; Vector3 dc = segment.Start  ray.Origin; if (Math.Abs(dc.Dot(da.Cross(db))) >= coPlanerThreshold) // Lines are not coplanar return false; double s = dc.Cross(db).Dot(da.Cross(db)) / da.Cross(db).LengthSquared; if (s >= 0.0 && s <= 1.0) // Means we have an intersection { Vector3 intersection = ray.Origin + s * da; // See if this lies on the segment if ((intersection  segment.Start).LengthSquared + (intersection  segment.End).LengthSquared <= segment.LengthSquared + lengthErrorThreshold) return true; } return false; } 