If your sister's mental stability has deteriorated to the point where she is unable to comprehend the obligation of prayers and fasting, she is not obligated to perform them according to Shariah. Therefore, it is not obligatory to provide fidyah for her missed prayers and fasts.
However, if your sister is sane enough to understand that prayers and fasting are "farz", but ends up making mistakes in her prayers and fasting, her prayers cannot be compensated for by giving fidyah. Her family members should help her in praying. One family member should sit with her and guide her through the steps of salah, such as rukoo and sujood. Alternatively, the other women in the house should pray together and encourage her to follow along.
As for fasting, if your sister experiences some improvement in her mental state during the month of Ramadan (meaning she is not unconscious or mentally disturbed to the point of madness for the entire month), she must perform qaza for the fasts she missed. If she is unable to perform qaza in her lifetime and requests her relatives to give fidyah on her behalf in her will, it is obligatory to give the fidyah.
The amount to be given as the fidyah for one fast is 1.75 kg of wheat or money equivalent to the price of 1.75 kg of wheat.