Effect of Initialization Variance on Layer Activations

\(\mathbf{x} \in \mathbb{R}^{100}\), \(x_i \sim \mathcal{N}(0,1)\), \(\Omega_{ij}^{(\ell)} \sim \mathcal{N}(0,\sigma_\Omega^2)\), \(\mathbf{h}_\ell = \mathrm{ReLU}\!\left(\Omega^{(\ell)} \mathbf{h}_{\ell-1}\right)\)