Graphics programming has this one very, very useful aspect, exponentially more valuable today: the matrix algebra pipelines, and then the requirement to 'think in matrix transforms' is a wonderful and visually engaging way to get your foundation for machine learning math.
I'm a data scientist and not a graphics programmer, but my guess is that it's just abstracted away more. If you're using ML/DL libraries, you're mostly just calling APIs that handle the linear algebra and calculus for you. Unless you're actively contributing to those libraries, you likely don't ever need to "touch" any of the underlying operations. Up to a point, it's useful to understand how things work under the hood, but where that point is kinda depends on your job. For instance, I could write code to do 'naive' matrix multiplication, but I couldn't, like, contribute to BLAS.
Both are large fields with many varied applications of linear algebra (and non-linear math too), and many people trying a lot of interesting & complicated ideas. The question is way too vague to answer meaningfully, it depends on what you mean by ‘graphics’ and by ‘machine learning’ and by ‘linear algebra’ and by ‘more complicated’. ;)
The linear algebra used in the basic raster pipeline to manage drawing a 3d unshaded mesh is pretty simple, and you can get by knowing just a little bit of linear algebra, like dot products and how to multiply matrices, and maybe what homogeneous coordinates are. But that is by no means the extent of linear algebra in all of graphics.
The linear algebra used in a basic neural network is also pretty simple, and you can get by knowing dot products and matrix multiply if you’re writing your own inference, and maybe just a tiny bit of derivative calculus if you’re writing your own backprop, but otherwise you don’t need anything else.
Students in both fields have to learn some basic linear algebra, but most people working in ML & graphics generally don’t use any linear algebra day to day, because most people aren’t writing inference/backprop and most people aren’t writing the graphics pipeline.
BTW, matrices and linear algebra are pure convenience for neural networks, and maybe for the graphics raster pipeline too. You can do both of these things without using matrices per se (though you might re-invent something equivalent and/or less efficient by avoiding matrices).
I mainly learned linear algebra via hands-on 3D graphics, and have a hard time thinking about a matrix as anything other than 4x4 and representing a linear transform...
How much do you even think about explicit matrix math when doing high-level ML?
I’ve not done high level ML, but I’ve done introductory ML and the truth is while the input space and the output space can have N and M dimensions, there’s not a lot of constraints involved. The matrix there are more randoms.
The whole ML field is basically about starting from random points and trying to find useful shapes and constraints. Basically like trying to get object likeness in clouds
3D graphics is so much more than the basic transforms. Add in all the deformation systems blending together, and those often being physics driven off the animation. You all have seen modern VFX, right? That is not basic 4x4 transforms.
there is absolutely no sense in which the SVD/PCA decomposition is just a rotation matrix. you should probably review your linear algebra textbook (hint: scaling is extremely important).
i don't understand who is having trouble reading the dialogue here you or i;
> there is absolutely no sense in which the SVD/PCA decomposition is just a rotation matrix... (hint: scaling is extremely important)
...
> SVD is the decomposition of a matrix into two rotation matrices and a scaling matrix, by definition:
yes that's exactly what i was implying when i said SVD more than just rotation, scaling is also important.
my point, which is my same original point, is that if you think learning about rotation/euler matrices is going to prepare you in any way, shape, or form for ML (vis-a-vis SVD/PCA or RoPE or anything else) you're in for a very rude awakening.
PCA is an orthogonal transformation of the covariance matrix, so like all orthogonal transformations, it’s _literally a rotation_ in N-dimensional space.
SVD is more complex but ultimately it’s just another useful decomposition of a matrix.
I’m not sure why you’re both negative and dismissive. Transformation matrices in graphics are a good and approachable way to get used to linear transformations, which turn out to be useful pretty much everywhere.
Whether or not that helps you with ML depends more on what you’re doing in ML. FAANG doesn’t have a monopoly on ML or on interesting work in ML.
> PCA is an orthogonal transformation of the covariance matrix
Yes you're now the second person the literally repeat the same thing I've already stated extremely clearly and succinctly: PCA is not just rotation (hint: you also need to understand covariance).
> I’m not sure why you’re both negative and dismissive. Transformation matrices in graphics are a good and approachable way to get used to linear transformations, which turn out to be useful pretty much everywhere.
I've already literally drawn the analogy/metaphor that I've drawn: if you think 2d/3d rotation matrices as they are used in graphics is any kind of introduction to the matrices in ML (modeling linear transformations or otherwise) then you're probably the type of person that believes that cash registers any kind of introduction to finance.
My point is not that hard to understand. Graphics in no way, way, shape, or form prepares you for ML. I don't understand why this is so controversial.
Have you done any serious graphics programming? Even at the OpenGL 1.x level? What you’re saying just doesn’t make sense.
Just because you’re rotating and translating things in 3-space doesn’t negate that you have a stack of transforms that relate a point in world space to one on screen space and you want to be able to project from one to the other.
Nor does it make it any easier when you need to think about how to stack transforms to achieve effects like rendering a mirror.
I honed a lot of useful practical skill with linear algebra trying to get graphics to do what I wanted. And I say this as someone who’s spent the bulk of my career using linear algebra in the context of quantum mechanics, physical simulation, and ML-adjacent areas.
> negate that you have a stack of transforms that relate a point in world space to one on screen space and you want to be able to project from one to the other.
no it doesn't "negate", it's all completely orthogonal (pun) or irrelevant. like for real just please take a look at
> tell me which operators you’re imagining have any resemblance with typical graphics linear algebra
FWIW, since it seems like you’re unaware: most of those are used in graphics in general, and have been used since long before Torch existed. Convolution’s extremely common. Pooling is just a type of image resampling to graphics people. Non-linear activations are just response functions that graphics people use for colors for example, also volume rendering. Normalization, linear, distance, vision, and shuffle layers are all absolutely standard common operations in graphics, on everything from images to meshes to volumes to matrices.
BTW, most of those Torch layers aren’t “linear algebra” per se, they are just convenient building blocks for neural networks, many of which are also convenient building blocks for graphics… and for similar reasons.
Was your point implicitly limited to rotations or a raster pipeline’s model-view-projection matrix? That certainly does not amount to all “graphics”, right?
> Graphics in no way, way, shape, or form prepares you for ML. I don't understand why this is so controversial.
This isn’t really controversial, it’s just not particularly true as stated. Graphics is much more than 3d rotation matrices, and doing real modern graphics involves all kinds of linear algebra, with immense amounts of overlap between the linear algebra that ML and computer vision use.
Perhaps missing from this conversation is any thoughtful consideration to the history of today’s ML and the cross pollination between the fields we call graphics, vision, and ML. The implicit assumption you seem to be making that they are distinct fields without a shared history and co-development and without a shared foundation is not a good assumption.
I personally know enough ex-graphics people that transitioned to ML and were well prepared by graphics and are wildly successful in ML that it makes your claim sound somewhat ignorant of what’s happened and is happening in both graphics and ML from my perspective, for what it’s worth.
... and I have been both situations for longer and have seen tons and tons of them (*)... So?
Not so hypotheticals -- Heck the inputs that you want labelled could be rotation matrices. The desired output could be a rotation matrix. Generating more convenient features could be via a rotation matrix. Dimensionality reduction could be through a reduction matrix. Sparsity could be encouraged by proper use of rotation matrices. Shows up if you want to build in group theoretic invariance in your predictive model.
A rotation matrix is but one of dozens and dozens of different types of basic transforms. It gets really fun with jacobian 12x12 matrix operations, and free form deformations. Which maps to ML far better than most imagine.
I don't really see this with modern graphics programming, but I was highly amused that my 1980s-1990s graphics skills (in particular, coordinate transform math) were very useful when I started working in robotics in the 2010s-2020s (because forward and inverse kinematics are exactly the same thing as 2d/3d projections.)
The trick there is that they both have related physical analogs, and machine learning math doesn't really (in that while you can visualize them spatially, it doesn't seem to help solve any problems in that space.)
I've been in ML for ~5 years in multiple FAANGs and I have never seen a rotation matrix.
The linear algebra used in the basic raster pipeline to manage drawing a 3d unshaded mesh is pretty simple, and you can get by knowing just a little bit of linear algebra, like dot products and how to multiply matrices, and maybe what homogeneous coordinates are. But that is by no means the extent of linear algebra in all of graphics.
The linear algebra used in a basic neural network is also pretty simple, and you can get by knowing dot products and matrix multiply if you’re writing your own inference, and maybe just a tiny bit of derivative calculus if you’re writing your own backprop, but otherwise you don’t need anything else.
Students in both fields have to learn some basic linear algebra, but most people working in ML & graphics generally don’t use any linear algebra day to day, because most people aren’t writing inference/backprop and most people aren’t writing the graphics pipeline.
BTW, matrices and linear algebra are pure convenience for neural networks, and maybe for the graphics raster pipeline too. You can do both of these things without using matrices per se (though you might re-invent something equivalent and/or less efficient by avoiding matrices).
How much do you even think about explicit matrix math when doing high-level ML?
The whole ML field is basically about starting from random points and trying to find useful shapes and constraints. Basically like trying to get object likeness in clouds
https://en.wikipedia.org/wiki/Singular_value_decomposition
> there is absolutely no sense in which the SVD/PCA decomposition is just a rotation matrix... (hint: scaling is extremely important)
...
> SVD is the decomposition of a matrix into two rotation matrices and a scaling matrix, by definition:
yes that's exactly what i was implying when i said SVD more than just rotation, scaling is also important.
my point, which is my same original point, is that if you think learning about rotation/euler matrices is going to prepare you in any way, shape, or form for ML (vis-a-vis SVD/PCA or RoPE or anything else) you're in for a very rude awakening.
> I've been in ML for ~5 years in multiple FAANGs and I have never seen a rotation matrix.
Presumably you've used SVD, but you've never seen a rotation matrix. So something is cooked.
Maybe corollary: that FAANG job wasn't that interesting.
SVD is more complex but ultimately it’s just another useful decomposition of a matrix.
I’m not sure why you’re both negative and dismissive. Transformation matrices in graphics are a good and approachable way to get used to linear transformations, which turn out to be useful pretty much everywhere.
Whether or not that helps you with ML depends more on what you’re doing in ML. FAANG doesn’t have a monopoly on ML or on interesting work in ML.
Yes you're now the second person the literally repeat the same thing I've already stated extremely clearly and succinctly: PCA is not just rotation (hint: you also need to understand covariance).
> I’m not sure why you’re both negative and dismissive. Transformation matrices in graphics are a good and approachable way to get used to linear transformations, which turn out to be useful pretty much everywhere.
I've already literally drawn the analogy/metaphor that I've drawn: if you think 2d/3d rotation matrices as they are used in graphics is any kind of introduction to the matrices in ML (modeling linear transformations or otherwise) then you're probably the type of person that believes that cash registers any kind of introduction to finance.
My point is not that hard to understand. Graphics in no way, way, shape, or form prepares you for ML. I don't understand why this is so controversial.
Have you done any serious graphics programming? Even at the OpenGL 1.x level? What you’re saying just doesn’t make sense.
Just because you’re rotating and translating things in 3-space doesn’t negate that you have a stack of transforms that relate a point in world space to one on screen space and you want to be able to project from one to the other.
Nor does it make it any easier when you need to think about how to stack transforms to achieve effects like rendering a mirror.
I honed a lot of useful practical skill with linear algebra trying to get graphics to do what I wanted. And I say this as someone who’s spent the bulk of my career using linear algebra in the context of quantum mechanics, physical simulation, and ML-adjacent areas.
no it doesn't "negate", it's all completely orthogonal (pun) or irrelevant. like for real just please take a look at
https://docs.pytorch.org/docs/2.12/nn.html
and tell me which operators you're imagining have any resemblance with typical graphics linear algebra.
like when you people make such claims do you really have anything concrete in mind or just hype?
FWIW, since it seems like you’re unaware: most of those are used in graphics in general, and have been used since long before Torch existed. Convolution’s extremely common. Pooling is just a type of image resampling to graphics people. Non-linear activations are just response functions that graphics people use for colors for example, also volume rendering. Normalization, linear, distance, vision, and shuffle layers are all absolutely standard common operations in graphics, on everything from images to meshes to volumes to matrices.
BTW, most of those Torch layers aren’t “linear algebra” per se, they are just convenient building blocks for neural networks, many of which are also convenient building blocks for graphics… and for similar reasons.
Was your point implicitly limited to rotations or a raster pipeline’s model-view-projection matrix? That certainly does not amount to all “graphics”, right?
> Graphics in no way, way, shape, or form prepares you for ML. I don't understand why this is so controversial.
This isn’t really controversial, it’s just not particularly true as stated. Graphics is much more than 3d rotation matrices, and doing real modern graphics involves all kinds of linear algebra, with immense amounts of overlap between the linear algebra that ML and computer vision use.
Perhaps missing from this conversation is any thoughtful consideration to the history of today’s ML and the cross pollination between the fields we call graphics, vision, and ML. The implicit assumption you seem to be making that they are distinct fields without a shared history and co-development and without a shared foundation is not a good assumption.
I personally know enough ex-graphics people that transitioned to ML and were well prepared by graphics and are wildly successful in ML that it makes your claim sound somewhat ignorant of what’s happened and is happening in both graphics and ML from my perspective, for what it’s worth.
If anyone needs a review it's not cognoboffin.
You led with the claim you have never seen a rotation matrix in ML. I am having doubts about whether you have the ability to recognise one.
I suspect new hires get a free pass as long as they can talk a storm about backpropagation these days.
Not so hypotheticals -- Heck the inputs that you want labelled could be rotation matrices. The desired output could be a rotation matrix. Generating more convenient features could be via a rotation matrix. Dimensionality reduction could be through a reduction matrix. Sparsity could be encouraged by proper use of rotation matrices. Shows up if you want to build in group theoretic invariance in your predictive model.
(*) If you consider Householders then even more
The trick there is that they both have related physical analogs, and machine learning math doesn't really (in that while you can visualize them spatially, it doesn't seem to help solve any problems in that space.)