Knot and link polynomials are topological invariants calculated from the expectation value of loop operators in topological field theories. In 3D Chern-Simons theory, these invariants can be found from crossing and braiding matrices of four-point conformal blocks of the boundary 2D CFT. We calculate crossing and braiding matrices for W_N conformal blocks with one component in the fundamental representation and another component in a rectangular representation of SU(N), which can be used to obtain HOMFLY knot and link invariants for these cases. We also discuss how our approach can be generalized to invariants in higher-representations of W_N algebra.