AUC ROC in keras is different when using tensorflow or scikit functions.
Two solutions for using AUC-ROC to train keras models, proposed here worked for me. But using tensorflow or scikit rocauc functions I get different results.
def auc(y_true, y_pred):
auc = tf.metrics.auc(y_true, y_pred)[1]
K.get_session().run(tf.local_variables_initializer())
return auc
and
def auc(y_true, y_pred):
return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)
Based on the history, it looks like both are being applied to train and validation.
When I plot history metrics, tensorflow curve looks very smoothed compared to scikit.
Shouldn't I get about the same results using both functions?
deep-learning keras scikit-learn performance
asked Dec 28 '18 at 16:40
aerijman
164
add a comment |
Two solutions for using AUC-ROC to train keras models, proposed here worked for me. But using tensorflow or scikit rocauc functions I get different results.
def auc(y_true, y_pred):
auc = tf.metrics.auc(y_true, y_pred)[1]
K.get_session().run(tf.local_variables_initializer())
return auc
and
def auc(y_true, y_pred):
return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)
Based on the history, it looks like both are being applied to train and validation.
When I plot history metrics, tensorflow curve looks very smoothed compared to scikit.
Shouldn't I get about the same results using both functions?
deep-learning keras scikit-learn performance
asked Dec 28 '18 at 16:40
aerijman
164
add a comment |
Two solutions for using AUC-ROC to train keras models, proposed here worked for me. But using tensorflow or scikit rocauc functions I get different results.
def auc(y_true, y_pred):
auc = tf.metrics.auc(y_true, y_pred)[1]
K.get_session().run(tf.local_variables_initializer())
return auc
and
def auc(y_true, y_pred):
return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)
Based on the history, it looks like both are being applied to train and validation.
When I plot history metrics, tensorflow curve looks very smoothed compared to scikit.
Shouldn't I get about the same results using both functions?
deep-learning keras scikit-learn performance
asked Dec 28 '18 at 16:40
aerijman
164
Two solutions for using AUC-ROC to train keras models, proposed here worked for me. But using tensorflow or scikit rocauc functions I get different results.
def auc(y_true, y_pred):
auc = tf.metrics.auc(y_true, y_pred)[1]
K.get_session().run(tf.local_variables_initializer())
return auc
and
def auc(y_true, y_pred):
return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)
Based on the history, it looks like both are being applied to train and validation.
When I plot history metrics, tensorflow curve looks very smoothed compared to scikit.
Shouldn't I get about the same results using both functions?
deep-learning keras scikit-learn performance
deep-learning keras scikit-learn performance
asked Dec 28 '18 at 16:40
aerijman
164
asked Dec 28 '18 at 16:40
aerijman
164
asked Dec 28 '18 at 16:40
aerijman
164
asked Dec 28 '18 at 16:40
aerijman
164
asked Dec 28 '18 at 16:40
aerijman
164
164
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
No, you shouldn't have the same numbers. All depends on the additional parameters:
tf.metrics.auc(
labels,
predictions,
weights=None,
num_thresholds=200,
metrics_collections=None,
updates_collections=None,
curve='ROC',
name=None,
summation_method='trapezoidal'
)
This means that this curve will have 200 points, so very smooth.
sklearn version doesn't have this kind of parameters:
roc_auc_score(y_true, y_score, average=’macro’, sample_weight=None, max_fpr=None)
The number of outputs depends on the curve and the number of points if I remember properly.
answered Dec 28 '18 at 16:48
Matthieu Brucher
51813
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f43247%2fauc-roc-in-keras-is-different-when-using-tensorflow-or-scikit-functions%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
No, you shouldn't have the same numbers. All depends on the additional parameters:
tf.metrics.auc(
labels,
predictions,
weights=None,
num_thresholds=200,
metrics_collections=None,
updates_collections=None,
curve='ROC',
name=None,
summation_method='trapezoidal'
)
This means that this curve will have 200 points, so very smooth.
sklearn version doesn't have this kind of parameters:
roc_auc_score(y_true, y_score, average=’macro’, sample_weight=None, max_fpr=None)
The number of outputs depends on the curve and the number of points if I remember properly.
answered Dec 28 '18 at 16:48
Matthieu Brucher
51813
add a comment |
No, you shouldn't have the same numbers. All depends on the additional parameters:
tf.metrics.auc(
labels,
predictions,
weights=None,
num_thresholds=200,
metrics_collections=None,
updates_collections=None,
curve='ROC',
name=None,
summation_method='trapezoidal'
)
This means that this curve will have 200 points, so very smooth.
sklearn version doesn't have this kind of parameters:
roc_auc_score(y_true, y_score, average=’macro’, sample_weight=None, max_fpr=None)
The number of outputs depends on the curve and the number of points if I remember properly.
answered Dec 28 '18 at 16:48
Matthieu Brucher
51813
add a comment |
No, you shouldn't have the same numbers. All depends on the additional parameters:
tf.metrics.auc(
labels,
predictions,
weights=None,
num_thresholds=200,
metrics_collections=None,
updates_collections=None,
curve='ROC',
name=None,
summation_method='trapezoidal'
)
This means that this curve will have 200 points, so very smooth.
sklearn version doesn't have this kind of parameters:
roc_auc_score(y_true, y_score, average=’macro’, sample_weight=None, max_fpr=None)
The number of outputs depends on the curve and the number of points if I remember properly.
answered Dec 28 '18 at 16:48
Matthieu Brucher
51813
No, you shouldn't have the same numbers. All depends on the additional parameters:
tf.metrics.auc(
labels,
predictions,
weights=None,
num_thresholds=200,
metrics_collections=None,
updates_collections=None,
curve='ROC',
name=None,
summation_method='trapezoidal'
)
This means that this curve will have 200 points, so very smooth.
sklearn version doesn't have this kind of parameters:
roc_auc_score(y_true, y_score, average=’macro’, sample_weight=None, max_fpr=None)
The number of outputs depends on the curve and the number of points if I remember properly.
answered Dec 28 '18 at 16:48
Matthieu Brucher
51813
answered Dec 28 '18 at 16:48
Matthieu Brucher
51813
answered Dec 28 '18 at 16:48
Matthieu Brucher
51813
answered Dec 28 '18 at 16:48
Matthieu Brucher
51813
51813
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f43247%2fauc-roc-in-keras-is-different-when-using-tensorflow-or-scikit-functions%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
