Assume {|u_{i}>} and {|t_{i}>} are orthonormal bases for a
Hilbert space.

Let
{|u_{i}>} = old basis,

{|t_{i}>} = new basis,

and let
U|u_{i}> = |t_{i}> be the unitary transformation from
the old
to new basis.

Then
<u_{i}|U|u_{j}> = <u_{i}|t_{j}> = U_{ij }= matrix elements of U.

Assume
|ψ> = ∑_{j}a_{j}|u_{j}> = ∑_{j}d_{j}|t_{j}>
is an arbitary vector in the vector space.

What is the relationship between the components of this vector in the two bases?

<t_{i}|ψ> =
<t_{i}|∑_{j}d_{j}|t_{j}> = d_{i}
=
∑_{j}a_{j}<t_{i}|u_{j}> = ∑_{j}U^{†}_{ij}a_{j}.

d_{i} = ∑_{j}U^{†}_{ij}a_{j }= components in new basis in terms of the components in the old basis.

a_{i} = ∑_{j}U_{ij}d_{j }=
components in old basis in terms of the components in the new basis.

Let |Φ> = Ω|ψ>,
|ψ> = ∑_{j}a_{j}|u_{j}>,
|Φ> = ∑_{j}a_{j}'|u_{j}>.

∑_{j}a_{j}'|u_{j}> = ∑_{j}a_{j}Ω|u_{j}>.

a_{i}' = ∑_{j}<u_{i}|Ω|u_{j}>a_{j} = ∑_{j}Ω_{ij}a_{j},

In a given representation, the matrix of Ω is defined by the action of Ω on the
basis vectors.

What is the relationship between the matrix elements of an operator Ω in the two bases
{|u_{i}>} and
{|t_{i}>}?

<t_{i}|Ω|t_{j}> = <t_{i}|UU^{†}ΩUU^{†}|t_{j}>
= <u_{i}|U^{†}ΩU|u_{j}>, or
Ω(new basis)_{ij} = (U^{†}ΩU)(old basis)_{ij}.

<u_{i}|Ω|u_{j}> = <u_{i}|U^{†}UΩU^{†}U|u_{j}>
= <t_{i}|UΩU^{†}|t_{j}>, or
Ω(old basis)_{ij} = (UΩU^{†})(new basis)_{ij}.

Assume that
Ω(old basis)_{ij} is given in the the
{|u_{i}>} basis, and it is not diagonal.

Assume
Ω(new basis)_{ij} is diagonal, the
{|t_{i}>} are eigenvectors of Ω.

To find the matrix of U we find the eigenvalues and expansion coefficient of the
eigenvectors of
Ω in the old basis. This is the typical eigenvalue problem.

Let a_{i}^{j} be the ith expansion coefficient of the jth
eigenvector in the
{|u_{i}>} bases.

In the
{|t_{i}>} the ith expansion coefficient of the jth eigenvector is just δ_{ij}.

Then
a_{i} = ∑_{k}U_{ik}d_{k} -->
a_{i}^{j} = ∑_{k}U_{ik}δ_{kj}
= U_{ij},
which shows that the columns of the matrix of U are the expansion coefficients
of the different eigenvectors in the
{|u_{i}>} basis, i.e. U_{ij} = a_{i}^{j}.

The matrix of Ω in the {|t_{i}>} basis is diagonal. It has the eigenvalues on its diagonal.

Note: What you call U and what you call U^{†} depends on which
basis you consider the old basis and which basis the new basis.

If the old basis is the eigenbasis and the new basis is a different basis, then
U and U^{†} are reversed.